A headline says that 52% of a town supports longer library hours. A second says 46%. Both come from honest surveys run in the same week. Neither pollster lied. One reached more young people than the other, or asked a slightly different question, or counted the "don't knows" differently, and the gap is the whole subject of this course in miniature.
Sociology cannot put a society in a laboratory. It has to ask, watch, count and compare, usually without being able to assign people to conditions. That makes method not a technicality but the argument: most disagreements between sociologists turn out to be disagreements about evidence. A graduate of this course reads a study from the bottom up, starting with who is in the data.
You cannot ask everyone, so you ask a sample and hope it stands for the rest. The mathematics of this is kind: a truly random sample of 1,000 people gives a margin of error near plus or minus 3 points, whatever the size of the country. The trouble is the word truly. You need a list of the population (a sampling frame), a chance mechanism for choosing from it, and answers from the people chosen. Failure at any of the three skews the picture, and a big sample does not cure a skewed one. The classic case is the Literary Digest poll of 1936, told in Elections, Parties & Public Opinion (35/15): millions of ballots, the wrong winner.
Today's version is nonresponse. Many surveys now hear back from a small share of those invited, and some kinds of people answer more than others. Pollsters repair this with weighting: if the census says 40% of adults are young but only 25% of respondents are, each young respondent counts for more than one person, and each older one for less. The usual weight is the group's share of the population divided by its share of the sample. Real surveys weight on several variables at once (age, sex, education, region), often by iterating until all the margins match, a method called raking.
An invented town: 40% of adults are young, 60% older. In truth, 70% of the young and 40% of the older support extending library hours (52% overall). Young people answer surveys less often. Set how often each group replies.
Illustrative data, not a real town. The first block of numbers is the long-run expectation; the second is one random draw from it, so it varies when you press the button. Weighting assumes that within each group the people who reply resemble those who do not, which is the assumption that cannot be checked from inside the survey.
Without JavaScript: with young people replying 20% of the time and older people 50%, a survey expects 46% support (the truth is 52%). Weighting the groups back to 40/60 recovers 52%.
Look at what you paid for the repair. Weighted answers vary more, because a few respondents now count for a lot: the effective sample size shrinks (Kish's formula, the square of the sum of the weights divided by the sum of the squares), and the margin of error widens. And the repair works only for what you weighted on. If the older people who reply are keener on libraries than the older people who do not, weighting by age cannot see it. Well-run surveys state their response rate, their weights and their frame. The long-running ones that sociologists lean on include the General Social Survey (United States, since 1972), the World Values Survey (since 1981), the European Social Survey (since 2002), and regional barometers such as Latinobarómetro and Afrobarometer (since the 1990s).
A question is a measuring device, and like any device it has a calibration. The same underlying opinion can give different readings with a different frame. Sociologists distinguish reliability (do repeated measurements agree?) from validity (are you measuring what you claim?), and they worry that the concept came first and the measure was an afterthought. Even "unemployed" is a decision: the International Labour Organization counts someone as unemployed if they have no work, are available, and have looked for it recently, which is not the same as being registered at an employment office, so two countries' rates can differ for definitional reasons.
The Pew Research Center, which publishes its question-wording experiments, offers clean examples. Pick one below.
Figures are from Pew Research Center's published question-wording experiments (see Sources for the year of each). Each pair was asked of comparable random samples.
Without JavaScript: 68% favoured military action in Iraq (January 2003) but 43% when casualties were mentioned; 51% favoured a law letting doctors give terminally ill patients "the means to end their lives" but 44% one saying "assist ... in committing suicide" (2005).
Three lessons recur. Context: mentioning a cost or consequence changes the answer, which is not always bias, since the second version may be the more informed one. Words with baggage: "welfare" and "assistance to the poor" do not draw the same support. Order: a question can colour the next one, so well-designed surveys randomise the order or put general questions before specific ones. The practical rule is to report the exact wording, and to treat any single percentage as the answer to that question, not to "what people think".
A survey is good at how many; an interview is good at what it means and why. In depth interviews, researchers ask open questions of a few dozen people, record and transcribe them, and then do the work that defines qualitative research: coding. They read the transcripts, label the passages with themes, and refine a codebook: a list of codes with definitions and examples. The aim is not a statistical estimate. It is an account of the mechanisms and categories in people's own terms, strong enough to be tested elsewhere. Traditions differ: grounded theory (Glaser and Strauss, 1967) builds the codes from the data upward, and thematic analysis (Braun and Clarke, 2006) is a widely taught step-by-step version.
How do you know two researchers would code alike? One answer is intercoder agreement: two people code the same passages independently and you compute how often they match after allowing for chance. The common statistic is Cohen's kappa (1960): agreement beyond chance, from 0 (no better than chance) to 1 (perfect). Try it. Ten invented snippets from people who moved to remote work, four codes, and a second coder whose choices are fixed.
Codebook. A = Autonomy (control over time or place). I = Isolation (missing people). C = Cost (money). B = Boundaries (work spilling into home).
Snippets are invented, not from real interviews. A real study would have far more text, and a first, messier codebook that the two coders revise together.
Without JavaScript: a snippet about wanting someone to notice when you struggle could be coded Autonomy or Isolation; one about affording rent by working from a village could be Cost or Autonomy. Codes overlap, and that is the point of the exercise.
Not everyone accepts kappa for this work. Researchers from the content-analysis tradition treat agreement figures as the mark of rigour. Braun and Clarke argue that in reflexive thematic analysis the researcher's interpretation is the instrument, so measuring agreement between coders misses what coding is for, and what matters is a clear account of how the themes were reached. Both camps agree on one thing: write down your rules, and say how many interviews you did and how you chose them.
Ethnography means spending long periods inside a setting: watching, joining in, taking field notes, and writing up what you saw. Bronisław Malinowski's Argonauts of the Western Pacific (1922), from years in the Trobriand Islands, set a standard in anthropology; William Foote Whyte's Street Corner Society (1943), from a Boston neighbourhood in the late 1930s, did so in sociology; and W. E. B. Du Bois's door-to-door work in Philadelphia (see Classical Social Theory, 35/75) combined counting with listening. The strength is meaning and mechanism: you see what people do, not just what they say they do. The weakness is generalisation: one site, one researcher, a presence that may change what is seen, and notes that no one else can read.
The best-known dispute of recent years shows what is at stake. Alice Goffman's On the Run (University of Chicago Press, 2014) described how young Black men in a poor Philadelphia neighbourhood lived under constant threat of arrest, drawing on years of fieldwork that began when she was an undergraduate. It was widely praised and won attention far outside sociology. Critics, among them the legal ethicist Steven Lubet and the journalist Jesse Singal, asked whether some episodes could be corroborated, whether she had reported accounts she was told as events she saw, and whether her description of her own role in one episode was accurate. Goffman replied in writing, and defended the book's purpose of letting marginalised people describe their own lives, rather than meeting an official-records test. An anonymous long critique that circulated on academic listservs was reviewed by her university, which found it without merit. Her adviser said he had checked some informants' identities. Other sociologists said they did not think she fabricated data but that heard accounts should have been presented as such. The honest summary is that the dispute stayed about how an ethnographer should handle evidence that cannot be checked, since the notes are confidential and protect people from arrest.
Participatory action research asks a different question: who is the research for? The Colombian sociologist Orlando Fals Borda (1925–2008), who helped found one of Latin America's first sociology faculties at the National University of Colombia in 1959 and wrote La Violencia en Colombia (1962) on a decade of political violence, worked in the 1970s and after with peasant and Caribbean-coast communities. He and others developed a method in which community members help choose the questions, collect the evidence and use the results, with knowledge returned to them. In his words, researchers should "not monopolise your knowledge nor impose arrogantly your technique." Supporters say insiders know which questions matter and catch what outsiders miss. Critics ask who counts as "the community", whether it can romanticise local views, and how conflicting views are handled. The idea has spread well beyond Colombia, in development, health and education research.
Experiments (see Research Methods in Psychology (35/19) and Research Methods in Political Science (35/18)) are the cleanest route from pattern to cause, and sociologists often cannot run them. Two things they can do are read data at the right level and look for accidents of history that act like experiments.
The first is a trap from Durkheim's time. In 1950 the sociologist William S. Robinson showed that in the 1930 US census the states with more foreign-born residents had lower illiteracy (a correlation near −0.5), while at the level of individuals, the foreign-born were somewhat more likely to be illiterate (about +0.12). The state pattern came from where immigrants settled, not from who among them could read. Inferring about individuals from group data is the ecological fallacy, and it hangs over any use of regional statistics. The tool below builds the reversal from scratch.
Six invented regions, with group A making up 5% to 90% of each. Inside every region, a person in A has a rate d points above someone not in A. But A members also settle where the background rate is lower.
Invented regions and rates: region rate = 40 + (d − c) × (share of A), in per cent. The point is the arithmetic of the reversal, not any real group.
Without JavaScript: with d = +5 and c = 20, group A's members have a rate 5 points higher in every region, yet regions with more of A have lower rates, a slope of −15 points across the full range of shares.
The second tool is the natural experiment: an event or rule that sorts people almost as if by lot. John Snow's study of cholera in London in the 1850s is the classic: households got water from two companies, one drawing from a sewage-polluted stretch of the Thames and one from cleaner water upriver, on streets where the two were mixed with no other pattern. Households on the first company's supply died of cholera at roughly eight times the rate in his best-known table for 1854 (315 against 37 deaths per 10,000 houses). The sociological examples are the same in kind: a draft lottery, a school-entry cut-off date, a lottery for housing vouchers. The randomised US Moving to Opportunity experiment (1994–98), which offered some families vouchers to move to lower-poverty neighbourhoods, found in a 2016 analysis by Raj Chetty, Nathaniel Hendren and Lawrence Katz that children who moved when young earned more as adults. Natural experiments are persuasive because the sorting wasn't chosen by the people, and limited because they answer narrow questions about narrow groups.
Sociology shares psychology's worries, with some differences. Much of its evidence is observational and comes from shared datasets, so the question is often not "does a repeat experiment reproduce it?" but "do different analysts get the same answer from the same data?" A striking test, led by Nate Breznau and colleagues (2022, in PNAS), gave 73 research teams the same cross-country survey data and the same question: does immigration reduce public support for social policy? The teams made reasonable but different choices about measures and models, and their estimates ran from large negative effects to large positive ones. A commentary pointed out that most results clustered near zero, which is a fair reading too. The lesson is not that the answer is arbitrary, but that a single published analysis understates the uncertainty.
For experiments the record is mixed in the usual way: a 2018 project by Colin Camerer and colleagues repeated 21 social-science experiments published in Nature and Science; about 13 of the 21 (62%) replicated in the original direction, with effects on average about half the original size. Field practice has shifted accordingly: pre-registered analyses, shared data and code, and journals that ask for them. Sociologists have also learned to separate three things a headline blurs: a robust description ("the gap is this large, by this measure"), a causal claim ("this produced that") and a normative claim ("so we should"). Methods can support the first firmly, the second with care, and the third not at all.
What sociology knows is shaped by how it asked. Know who is in the data and how they were weighted, treat a percentage as the answer to a particular question, code interviews with rules you can show, use ethnography and participatory work for meaning and for whose question it is, read group data and individual data separately, and expect different analysts to differ.
Try this: the next time you see a poll or a "study shows" headline, find the exact question wording, the sample size and who was sampled. If you cannot find all three, treat the number as a lead, not a fact.
Sources. Pew Research Center, "Writing Survey Questions" (open versus closed questions; Iraq, January 2003: 68% favour and 25% oppose against 43% and 48% with casualties; end-of-life wording, 2005: 51% against 44%; "domestic policy or foreign policy" January 2002: 52% and 34% against 33% and 52% with "the war on terrorism"; December 2008 satisfaction question: 88% dissatisfied after the Bush approval item against 78% without; 2008 vote issue, closed 58% against 35% open-ended). Kish on effective sample size (design effect from unequal weights). Glaser and Strauss, The Discovery of Grounded Theory (1967); Braun and Clarke, "Using thematic analysis in psychology," Qualitative Research in Psychology 2006; Cohen, "A coefficient of agreement for nominal scales," Educational and Psychological Measurement 1960. Malinowski 1922; Whyte 1943. Goffman, On the Run (2014): the account of the dispute follows the reporting summarised on Wikipedia (Lubet, Singal, the University of Wisconsin–Madison review, Duneier's statement), and the book itself; I have not read the unpublished critique. Fals Borda (1925–2008), La Violencia en Colombia (1962), the 1959 faculty, and the quoted line as given on Wikipedia, "Orlando Fals Borda". Robinson, "Ecological correlations and the behavior of individuals," American Sociological Review 1950 (−0.53 and +0.12 as reported; later corrected to about −0.46, per Wikipedia, "Ecological fallacy"). Snow, On the Mode of Communication of Cholera (1855), with the "about fourteen times" figure from Wikipedia, "John Snow". Chetty, Hendren and Katz, American Economic Review 2016. Breznau et al., PNAS 119(44), 2022. Camerer et al., Nature Human Behaviour 2018. ILO definition of unemployment. Start years of surveys are from each programme's own site. The sampling, coding and region numbers were run in node. Study descriptions are for teaching; this is not advice.